Client performance telemetry: always-on instrument, Loki ingest, and Grafana dashboard - #5597
Merged
Conversation
Accept batched client performance beacons (authenticated by the realm-server session token) and re-emit each event as a single structured JSON log line on the boxel:client-perf channel, which the existing stdout -> alloy -> Loki pipeline carries to Grafana. The handler is schema-agnostic beyond an envelope check: it caps body size (413) and event count (400), and the trusted envelope fields (channel, authenticated user id) are applied last so a per-event field cannot spoof them. Serve Document-Policy: js-profiling on the host HTML so the client telemetry's sampled JS self-profiler can capture wedge stacks. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A passive singleton instrument that measures client-side performance and ships signal-gated, batched events to the realm-server /_client-telemetry endpoint. It records six event types: server-request latency (correlated to server logs via the logging correlation id), response deserialization cost, card-to- interactive latency (with card-id attribution), main-thread wedges (a heartbeat gap of 10s or more, with long-animation-frame / longtask attribution and, on sampled sessions, JS self-profiler stacks), loader/store rebuilds (with the invalidated trigger module), and per-realm-event reload churn. The instrument is disabled under tests (opt-in via enableForTest) and inside a prerender tab, feature-detects every web API it uses, and tears down every timer, observer, and listener. It stays quiet while the tab is hidden (throttled timers are not wedges), bounds each flush to the server's caps, and samples the JS self-profiler per session so continuous profiling cost is not paid by every client. Every hook is internally guarded so a telemetry failure never disturbs the path that emitted the event. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Visualizes the boxel:client-perf telemetry from Loki: an overview row of headline health, a collapsible row per event type, a by-user row surfacing which accounts are hitting issues (wedges, worst card-to-interactive, and signal volume per user), and a wedge breakdown that names the blocking frame (function plus source URL and character offset, resolvable via source map) and lists the worst blocking scripts per wedge. A wedge is labelled throughout as a main-thread freeze of 10s or more. Every panel is driven by shared env / user / session template variables so selecting a user or session drills the whole board into one client. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Observability diff (vs staging)Diff truncated (104770 bytes; limit 60000). Full diff: https://github.com/cardstack/boxel/actions/runs/30282158533 Show diffdiff --git a/tmp/committed-canon.GuifT7/dashboards/boxel-status/client-performance.json b/tmp/committed-canon.GuifT7/dashboards/boxel-status/client-performance.json
new file mode 100644
index 0000000..582cd2c
--- /dev/null
+++ b/tmp/committed-canon.GuifT7/dashboards/boxel-status/client-performance.json
@@ -0,0 +1,3067 @@
+{
+ "apiVersion": "dashboard.grafana.app/v1beta1",
+ "kind": "Dashboard",
+ "metadata": {
+ "annotations": {
+ "grafana.app/folder": "defd2d156sav4d"
+ },
+ "name": "boxel-client-perf"
+ },
+ "spec": {
+ "annotations": {
+ "list": [
+ {
+ "builtIn": 1,
+ "datasource": {
+ "type": "grafana",
+ "uid": "-- Grafana --"
+ },
+ "enable": true,
+ "hide": true,
+ "iconColor": "rgba(0, 211, 255, 1)",
+ "name": "Annotations & Alerts",
+ "type": "dashboard"
+ }
+ ]
+ },
+ "description": "Client-side performance telemetry (channel boxel:client-perf) re-emitted by realm-server as structured JSON log lines and read from Loki. Overview headline signals plus a collapsible row per event type; the User/Session variables drill the whole board into a single client.",
+ "editable": true,
+ "fiscalYearStartMonth": 0,
+ "graphTooltip": 0,
+ "links": [],
+ "panels": [
+ {
+ "collapsed": false,
+ "gridPos": {
+ "h": 1,
+ "w": 24,
+ "x": 0,
+ "y": 0
+ },
+ "id": 2,
+ "panels": [],
+ "title": "Overview",
+ "type": "row"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Distinct client sessions that emitted any boxel:client-perf event in the selected time range. Drill filters (user/session) narrow this.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "thresholds"
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "blue"
+ },
+ {
+ "color": "green",
+ "value": 1
+ }
+ ]
+ },
+ "unit": "short"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 4,
+ "w": 6,
+ "x": 0,
+ "y": 1
+ },
+ "id": 3,
+ "options": {
+ "colorMode": "value",
+ "graphMode": "area",
+ "justifyMode": "center",
+ "orientation": "auto",
+ "reduceOptions": {
+ "calcs": [
+ "lastNotNull"
+ ],
+ "fields": "",
+ "values": false
+ },
+ "textMode": "auto"
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "count(count by (session_id) (count_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" [$__range])))",
+ "queryType": "instant",
+ "refId": "A"
+ }
+ ],
+ "title": "Active sessions",
+ "type": "stat"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Count of card-load events in the selected range.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "thresholds"
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ }
+ ]
+ },
+ "unit": "short"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 4,
+ "w": 6,
+ "x": 6,
+ "y": 1
+ },
+ "id": 4,
+ "options": {
+ "colorMode": "value",
+ "graphMode": "area",
+ "justifyMode": "center",
+ "orientation": "auto",
+ "reduceOptions": {
+ "calcs": [
+ "lastNotNull"
+ ],
+ "fields": "",
+ "values": false
+ },
+ "textMode": "auto"
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "sum(count_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"card-load\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" [$__range]))",
+ "queryType": "instant",
+ "refId": "A"
+ }
+ ],
+ "title": "Card loads (range)",
+ "type": "stat"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Count of wedge events (main-thread stalls) in the selected range. Non-zero is worth investigating in the wedge row below. A wedge is a main-thread freeze of 10s or more (the 100ms heartbeat missed a tick by ≥10s). Sub-second jank is not counted here; see a keepalive's max_gap_ms for that.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "thresholds"
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ },
+ {
+ "color": "yellow",
+ "value": 1
+ },
+ {
+ "color": "red",
+ "value": 10
+ }
+ ]
+ },
+ "unit": "short"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 4,
+ "w": 6,
+ "x": 12,
+ "y": 1
+ },
+ "id": 5,
+ "options": {
+ "colorMode": "value",
+ "graphMode": "area",
+ "justifyMode": "center",
+ "orientation": "auto",
+ "reduceOptions": {
+ "calcs": [
+ "lastNotNull"
+ ],
+ "fields": "",
+ "values": false
+ },
+ "textMode": "auto"
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "sum(count_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"wedge\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" [$__range]))",
+ "queryType": "instant",
+ "refId": "A"
+ }
+ ],
+ "title": "Wedges ≥10s (range)",
+ "type": "stat"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Count of rebuild events (loader/module refetch cycles) in the selected range.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "thresholds"
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ },
+ {
+ "color": "yellow",
+ "value": 5
+ }
+ ]
+ },
+ "unit": "short"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 4,
+ "w": 6,
+ "x": 18,
+ "y": 1
+ },
+ "id": 6,
+ "options": {
+ "colorMode": "value",
+ "graphMode": "area",
+ "justifyMode": "center",
+ "orientation": "auto",
+ "reduceOptions": {
+ "calcs": [
+ "lastNotNull"
+ ],
+ "fields": "",
+ "values": false
+ },
+ "textMode": "auto"
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "sum(count_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"rebuild\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" [$__range]))",
+ "queryType": "instant",
+ "refId": "A"
+ }
+ ],
+ "title": "Rebuilds (range)",
+ "type": "stat"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "p95 card-to-interactive latency (card-load settle_ms) alongside p95 server-request duration_ms. The headline responsiveness signals.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "palette-classic"
+ },
+ "custom": {
+ "axisLabel": "ms",
+ "axisPlacement": "auto",
+ "barAlignment": 0,
+ "drawStyle": "line",
+ "fillOpacity": 10,
+ "gradientMode": "none",
+ "hideFrom": {
+ "legend": false,
+ "tooltip": false,
+ "viz": false
+ },
+ "lineInterpolation": "linear",
+ "lineWidth": 1,
+ "pointSize": 5,
+ "scaleDistribution": {
+ "type": "linear"
+ },
+ "showPoints": "never",
+ "spanNulls": false,
+ "stacking": {
+ "group": "A",
+ "mode": "none"
+ },
+ "thresholdsStyle": {
+ "mode": "off"
+ }
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ }
+ ]
+ },
+ "unit": "ms"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 0,
+ "y": 5
+ },
+ "id": 7,
+ "options": {
+ "legend": {
+ "calcs": [
+ "mean",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "quantile_over_time(0.95, {service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"card-load\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap settle_ms [$__interval]) by ()",
+ "legendFormat": "card settle p95",
+ "queryType": "range",
+ "refId": "A"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "quantile_over_time(0.95, {service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"server-request\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap duration_ms [$__interval]) by ()",
+ "legendFormat": "server-request p95",
+ "queryType": "range",
+ "refId": "B"
+ }
+ ],
+ "title": "Interactivity p95 (ms)",
+ "type": "timeseries"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Rate of the signal-gated event types that flag anomalies. wedge and rebuild spikes correlate with degraded client sessions.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "palette-classic"
+ },
+ "custom": {
+ "axisLabel": "events",
+ "axisPlacement": "auto",
+ "barAlignment": 0,
+ "drawStyle": "line",
+ "fillOpacity": 10,
+ "gradientMode": "none",
+ "hideFrom": {
+ "legend": false,
+ "tooltip": false,
+ "viz": false
+ },
+ "lineInterpolation": "linear",
+ "lineWidth": 1,
+ "pointSize": 5,
+ "scaleDistribution": {
+ "type": "linear"
+ },
+ "showPoints": "never",
+ "spanNulls": false,
+ "stacking": {
+ "group": "A",
+ "mode": "none"
+ },
+ "thresholdsStyle": {
+ "mode": "off"
+ }
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ }
+ ]
+ },
+ "unit": "short"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 12,
+ "y": 5
+ },
+ "id": 8,
+ "options": {
+ "legend": {
+ "calcs": [
+ "mean",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "sum(count_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"wedge\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" [$__interval]))",
+ "legendFormat": "wedge",
+ "queryType": "range",
+ "refId": "A"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "sum(count_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"rebuild\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" [$__interval]))",
+ "legendFormat": "rebuild",
+ "queryType": "range",
+ "refId": "B"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "sum(count_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"realm-event\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" [$__interval]))",
+ "legendFormat": "realm-event",
+ "queryType": "range",
+ "refId": "C"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "sum(count_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"card-load\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" [$__interval]))",
+ "legendFormat": "card-load",
+ "queryType": "range",
+ "refId": "D"
+ }
+ ],
+ "title": "Signal event rate (events / interval)",
+ "type": "timeseries"
+ },
+ {
+ "collapsed": true,
+ "gridPos": {
+ "h": 1,
+ "w": 24,
+ "x": 0,
+ "y": 14
+ },
+ "id": 13,
+ "panels": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Card-to-interactive time (settle_ms): p50/p95/p99 across all card-load events in each interval.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "palette-classic"
+ },
+ "custom": {
+ "axisLabel": "ms",
+ "axisPlacement": "auto",
+ "barAlignment": 0,
+ "drawStyle": "line",
+ "fillOpacity": 10,
+ "gradientMode": "none",
+ "hideFrom": {
+ "legend": false,
+ "tooltip": false,
+ "viz": false
+ },
+ "lineInterpolation": "linear",
+ "lineWidth": 1,
+ "pointSize": 5,
+ "scaleDistribution": {
+ "type": "linear"
+ },
+ "showPoints": "never",
+ "spanNulls": false,
+ "stacking": {
+ "group": "A",
+ "mode": "none"
+ },
+ "thresholdsStyle": {
+ "mode": "off"
+ }
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ }
+ ]
+ },
+ "unit": "ms"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 0,
+ "y": 15
+ },
+ "id": 9,
+ "options": {
+ "legend": {
+ "calcs": [
+ "mean",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "quantile_over_time(0.50, {service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"card-load\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap settle_ms [$__interval]) by ()",
+ "legendFormat": "p50",
+ "queryType": "range",
+ "refId": "A"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "quantile_over_time(0.95, {service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"card-load\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap settle_ms [$__interval]) by ()",
+ "legendFormat": "p95",
+ "queryType": "range",
+ "refId": "B"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "quantile_over_time(0.99, {service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"card-load\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap settle_ms [$__interval]) by ()",
+ "legendFormat": "p99",
+ "queryType": "range",
+ "refId": "C"
+ }
+ ],
+ "title": "card-load settle_ms percentiles",
+ "type": "timeseries"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Network/loading time (loading_ms): p50/p95/p99 across all card-load events in each interval.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "palette-classic"
+ },
+ "custom": {
+ "axisLabel": "ms",
+ "axisPlacement": "auto",
+ "barAlignment": 0,
+ "drawStyle": "line",
+ "fillOpacity": 10,
+ "gradientMode": "none",
+ "hideFrom": {
+ "legend": false,
+ "tooltip": false,
+ "viz": false
+ },
+ "lineInterpolation": "linear",
+ "lineWidth": 1,
+ "pointSize": 5,
+ "scaleDistribution": {
+ "type": "linear"
+ },
+ "showPoints": "never",
+ "spanNulls": false,
+ "stacking": {
+ "group": "A",
+ "mode": "none"
+ },
+ "thresholdsStyle": {
+ "mode": "off"
+ }
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ }
+ ]
+ },
+ "unit": "ms"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 12,
+ "y": 15
+ },
+ "id": 10,
+ "options": {
+ "legend": {
+ "calcs": [
+ "mean",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "quantile_over_time(0.50, {service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"card-load\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap loading_ms [$__interval]) by ()",
+ "legendFormat": "p50",
+ "queryType": "range",
+ "refId": "A"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "quantile_over_time(0.95, {service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"card-load\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap loading_ms [$__interval]) by ()",
+ "legendFormat": "p95",
+ "queryType": "range",
+ "refId": "B"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "quantile_over_time(0.99, {service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"card-load\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap loading_ms [$__interval]) by ()",
+ "legendFormat": "p99",
+ "queryType": "range",
+ "refId": "C"
+ }
+ ],
+ "title": "card-load loading_ms percentiles",
+ "type": "timeseries"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Top 20 card_id values by worst observed settle_ms over the selected range. Instant query; widen the time range to survey more history.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "thresholds"
+ },
+ "custom": {
+ "align": "auto",
+ "cellOptions": {
+ "type": "auto"
+ },
+ "filterable": true,
+ "inspect": false
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ },
+ {
+ "color": "red",
+ "value": 80
+ }
+ ]
+ }
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 9,
+ "w": 12,
+ "x": 0,
+ "y": 23
+ },
+ "id": 11,
+ "options": {
+ "cellHeight": "sm",
+ "footer": {
+ "countRows": false,
+ "fields": "",
+ "reducer": [
+ "sum"
+ ],
+ "show": false
+ },
+ "showHeader": true
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "topk(20, max by (card_id) (max_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"card-load\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap settle_ms [$__range])))",
+ "legendFormat": "",
+ "queryType": "instant",
+ "refId": "A"
+ }
+ ],
+ "title": "Slowest card loads by card_id (max settle_ms)",
+ "transformations": [
+ {
+ "id": "labelsToFields",
+ "options": {}
+ },
+ {
+ "id": "merge",
+ "options": {}
+ },
+ {
+ "id": "organize",
+ "options": {
+ "excludeByName": {
+ "Time": true
+ },
+ "renameByName": {
+ "Value #A": "max settle_ms"
+ }
+ }
+ }
+ ],
+ "type": "table"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Card-load event rate per interval, plus the average num_loads (link/dependency loads) per card-load event.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "palette-classic"
+ },
+ "custom": {
+ "axisLabel": "",
+ "axisPlacement": "auto",
+ "barAlignment": 0,
+ "drawStyle": "line",
+ "fillOpacity": 10,
+ "gradientMode": "none",
+ "hideFrom": {
+ "legend": false,
+ "tooltip": false,
+ "viz": false
+ },
+ "lineInterpolation": "linear",
+ "lineWidth": 1,
+ "pointSize": 5,
+ "scaleDistribution": {
+ "type": "linear"
+ },
+ "showPoints": "never",
+ "spanNulls": false,
+ "stacking": {
+ "group": "A",
+ "mode": "none"
+ },
+ "thresholdsStyle": {
+ "mode": "off"
+ }
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ }
+ ]
+ },
+ "unit": "short"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 9,
+ "w": 12,
+ "x": 12,
+ "y": 23
+ },
+ "id": 12,
+ "options": {
+ "legend": {
+ "calcs": [
+ "mean",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "sum(count_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"card-load\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" [$__interval]))",
+ "legendFormat": "card-loads",
+ "queryType": "range",
+ "refId": "A"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "avg_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"card-load\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap num_loads [$__interval]) by ()",
+ "legendFormat": "avg num_loads",
+ "queryType": "range",
+ "refId": "B"
+ }
+ ],
+ "title": "card-load rate & avg num_loads",
+ "type": "timeseries"
+ }
+ ],
+ "title": "card-load",
+ "type": "row"
+ },
+ {
+ "collapsed": true,
+ "gridPos": {
+ "h": 1,
+ "w": 24,
+ "x": 0,
+ "y": 24
+ },
+ "id": 19,
+ "panels": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "p95 duration_ms grouped by endpoint. Isolates which realm-server endpoints dominate client-observed request latency.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "palette-classic"
+ },
+ "custom": {
+ "axisLabel": "ms",
+ "axisPlacement": "auto",
+ "barAlignment": 0,
+ "drawStyle": "line",
+ "fillOpacity": 10,
+ "gradientMode": "none",
+ "hideFrom": {
+ "legend": false,
+ "tooltip": false,
+ "viz": false
+ },
+ "lineInterpolation": "linear",
+ "lineWidth": 1,
+ "pointSize": 5,
+ "scaleDistribution": {
+ "type": "linear"
+ },
+ "showPoints": "never",
+ "spanNulls": false,
+ "stacking": {
+ "group": "A",
+ "mode": "none"
+ },
+ "thresholdsStyle": {
+ "mode": "off"
+ }
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ }
+ ]
+ },
+ "unit": "ms"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 0,
+ "y": 25
+ },
+ "id": 14,
+ "options": {
+ "legend": {
+ "calcs": [
+ "mean",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "quantile_over_time(0.95, {service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"server-request\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap duration_ms [$__interval]) by (endpoint)",
+ "legendFormat": "{{endpoint}}",
+ "queryType": "range",
+ "refId": "A"
+ }
+ ],
+ "title": "server-request duration p95 by endpoint",
+ "type": "timeseries"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "p50/p95/p99 of duration_ms across all server-request events in each interval.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "palette-classic"
+ },
+ "custom": {
+ "axisLabel": "ms",
+ "axisPlacement": "auto",
+ "barAlignment": 0,
+ "drawStyle": "line",
+ "fillOpacity": 10,
+ "gradientMode": "none",
+ "hideFrom": {
+ "legend": false,
+ "tooltip": false,
+ "viz": false
+ },
+ "lineInterpolation": "linear",
+ "lineWidth": 1,
+ "pointSize": 5,
+ "scaleDistribution": {
+ "type": "linear"
+ },
+ "showPoints": "never",
+ "spanNulls": false,
+ "stacking": {
+ "group": "A",
+ "mode": "none"
+ },
+ "thresholdsStyle": {
+ "mode": "off"
+ }
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ }
+ ]
+ },
+ "unit": "ms"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 12,
+ "y": 25
+ },
+ "id": 15,
+ "options": {
+ "legend": {
+ "calcs": [
+ "mean",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "quantile_over_time(0.50, {service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"server-request\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap duration_ms [$__interval]) by ()",
+ "legendFormat": "p50",
+ "queryType": "range",
+ "refId": "A"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "quantile_over_time(0.95, {service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"server-request\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap duration_ms [$__interval]) by ()",
+ "legendFormat": "p95",
+ "queryType": "range",
+ "refId": "B"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "quantile_over_time(0.99, {service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"server-request\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap duration_ms [$__interval]) by ()",
+ "legendFormat": "p99",
+ "queryType": "range",
+ "refId": "C"
+ }
+ ],
+ "title": "server-request duration percentiles (all endpoints)",
+ "type": "timeseries"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Rate of 4xx/5xx server-request responses grouped by status code.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "palette-classic"
+ },
+ "custom": {
+ "axisLabel": "errors",
+ "axisPlacement": "auto",
+ "barAlignment": 0,
+ "drawStyle": "line",
+ "fillOpacity": 10,
+ "gradientMode": "none",
+ "hideFrom": {
+ "legend": false,
+ "tooltip": false,
+ "viz": false
+ },
+ "lineInterpolation": "linear",
+ "lineWidth": 1,
+ "pointSize": 5,
+ "scaleDistribution": {
+ "type": "linear"
+ },
+ "showPoints": "never",
+ "spanNulls": false,
+ "stacking": {
+ "group": "A",
+ "mode": "none"
+ },
+ "thresholdsStyle": {
+ "mode": "off"
+ }
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ }
+ ]
+ },
+ "unit": "short"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 0,
+ "y": 33
+ },
+ "id": 16,
+ "options": {
+ "legend": {
+ "calcs": [
+ "mean",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "sum by (status) (count_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"server-request\" | status=~\"[45]..\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" [$__interval]))",
+ "legendFormat": "{{status}}",
+ "queryType": "range",
+ "refId": "A"
+ }
+ ],
+ "title": "server-request error rate by status",
+ "type": "timeseries"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Per-endpoint worst-case duration_ms (query A) and retried-request count (query B) over the selected range. Instant queries merged on endpoint.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "thresholds"
+ },
+ "custom": {
+ "align": "auto",
+ "cellOptions": {
+ "type": "auto"
+ },
+ "filterable": true,
+ "inspect": false
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ },
+ {
+ "color": "red",
+ "value": 80
+ }
+ ]
+ }
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 12,
+ "y": 33
+ },
+ "id": 17,
+ "options": {
+ "cellHeight": "sm",
+ "footer": {
+ "countRows": false,
+ "fields": "",
+ "reducer": [
+ "sum"
+ ],
+ "show": false
+ },
+ "showHeader": true
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "topk(15, max by (endpoint) (max_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"server-request\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap duration_ms [$__range])))",
+ "legendFormat": "",
+ "queryType": "instant",
+ "refId": "A"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "sum by (endpoint) (count_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"server-request\" | retried=\"true\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" [$__range]))",
+ "legendFormat": "",
+ "queryType": "instant",
+ "refId": "B"
+ }
+ ],
+ "title": "Slowest & most-retried endpoints",
+ "transformations": [
+ {
+ "id": "labelsToFields",
+ "options": {}
+ },
+ {
+ "id": "merge",
+ "options": {}
+ },
+ {
+ "id": "organize",
+ "options": {
+ "excludeByName": {
+ "Time": true
+ },
+ "renameByName": {
+ "Value #A": "max duration_ms",
+ "Value #B": "retried"
+ }
+ }
+ }
+ ],
+ "type": "table"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Raw server-request telemetry lines, newest first. Each carries correlation_id for cross-referencing realm-server request logs.",
+ "gridPos": {
+ "h": 10,
+ "w": 24,
+ "x": 0,
+ "y": 41
+ },
+ "id": 18,
+ "options": {
+ "dedupStrategy": "none",
+ "enableLogDetails": true,
+ "prettifyLogMessage": false,
+ "showCommonLabels": false,
+ "showLabels": false,
+ "showTime": true,
+ "sortOrder": "Descending",
+ "wrapLogMessage": true
+ },
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "{service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"server-request\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\"",
+ "queryType": "range",
+ "refId": "A"
+ }
+ ],
+ "title": "server-request log lines (correlation_id)",
+ "type": "logs"
+ }
+ ],
+ "title": "server-request",
+ "type": "row"
+ },
+ {
+ "collapsed": true,
+ "gridPos": {
+ "h": 1,
+ "w": 24,
+ "x": 0,
+ "y": 42
+ },
+ "id": 22,
+ "panels": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "p95 deserialize duration_ms grouped by card_type. Surfaces card types whose document deserialization is expensive.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "palette-classic"
+ },
+ "custom": {
+ "axisLabel": "ms",
+ "axisPlacement": "auto",
+ "barAlignment": 0,
+ "drawStyle": "line",
+ "fillOpacity": 10,
+ "gradientMode": "none",
+ "hideFrom": {
+ "legend": false,
+ "tooltip": false,
+ "viz": false
+ },
+ "lineInterpolation": "linear",
+ "lineWidth": 1,
+ "pointSize": 5,
+ "scaleDistribution": {
+ "type": "linear"
+ },
+ "showPoints": "never",
+ "spanNulls": false,
+ "stacking": {
+ "group": "A",
+ "mode": "none"
+ },
+ "thresholdsStyle": {
+ "mode": "off"
+ }
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ }
+ ]
+ },
+ "unit": "ms"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 0,
+ "y": 43
+ },
+ "id": 20,
+ "options": {
+ "legend": {
+ "calcs": [
+ "mean",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "quantile_over_time(0.95, {service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"deserialize\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap duration_ms [$__interval]) by (card_type)",
+ "legendFormat": "{{card_type}}",
+ "queryType": "range",
+ "refId": "A"
+ }
+ ],
+ "title": "deserialize duration p95 by card_type",
+ "type": "timeseries"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Per card_type: avg duration_ms (A), avg doc_bytes (B), avg included_count (C) over the range. Intended as a cost table to spot large-document / high-fan-in card types (a scatter of duration vs size).",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "thresholds"
+ },
+ "custom": {
+ "align": "auto",
+ "cellOptions": {
+ "type": "auto"
+ },
+ "filterable": true,
+ "inspect": false
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ },
+ {
+ "color": "red",
+ "value": 80
+ }
+ ]
+ }
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 12,
+ "y": 43
+ },
+ "id": 21,
+ "options": {
+ "cellHeight": "sm",
+ "footer": {
+ "countRows": false,
+ "fields": "",
+ "reducer": [
+ "sum"
+ ],
+ "show": false
+ },
+ "showHeader": true
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "avg by (card_type) (avg_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"deserialize\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap duration_ms [$__range]))",
+ "legendFormat": "",
+ "queryType": "instant",
+ "refId": "A"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "avg by (card_type) (avg_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"deserialize\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap doc_bytes [$__range]))",
+ "legendFormat": "",
+ "queryType": "instant",
+ "refId": "B"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "avg by (card_type) (avg_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"deserialize\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap included_count [$__range]))",
+ "legendFormat": "",
+ "queryType": "instant",
+ "refId": "C"
+ }
+ ],
+ "title": "deserialize cost by card_type (duration vs bytes vs included)",
+ "transformations": [
+ {
+ "id": "labelsToFields",
+ "options": {}
+ },
+ {
+ "id": "merge",
+ "options": {}
+ },
+ {
+ "id": "organize",
+ "options": {
+ "excludeByName": {
+ "Time": true
+ },
+ "renameByName": {
+ "Value #A": "avg duration_ms",
+ "Value #B": "avg doc_bytes",
+ "Value #C": "avg included_count"
+ }
+ }
+ }
+ ],
+ "type": "table"
+ }
+ ],
+ "title": "deserialize",
+ "type": "row"
+ },
+ {
+ "collapsed": true,
+ "description": "A wedge is a main-thread freeze of 10s or more (the 100ms heartbeat missed a tick by ≥10s). Sub-second jank is not counted here; see a keepalive's max_gap_ms for that.",
+ "gridPos": {
+ "h": 1,
+ "w": 24,
+ "x": 0,
+ "y": 44
+ },
+ "id": 25,
+ "panels": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Wedge event rate per interval (left axis) and the worst blocked_ms observed per interval (right axis, ms). A wedge is a main-thread stall; blocked_ms is how long the thread was blocked. A wedge is a main-thread freeze of 10s or more (the 100ms heartbeat missed a tick by ≥10s). Sub-second jank is not counted here; see a keepalive's max_gap_ms for that.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "palette-classic"
+ },
+ "custom": {
+ "axisLabel": "wedges",
+ "axisPlacement": "auto",
+ "barAlignment": 0,
+ "drawStyle": "line",
+ "fillOpacity": 10,
+ "gradientMode": "none",
+ "hideFrom": {
+ "legend": false,
+ "tooltip": false,
+ "viz": false
+ },
+ "lineInterpolation": "linear",
+ "lineWidth": 1,
+ "pointSize": 5,
+ "scaleDistribution": {
+ "type": "linear"
+ },
+ "showPoints": "never",
+ "spanNulls": false,
+ "stacking": {
+ "group": "A",
+ "mode": "none"
+ },
+ "thresholdsStyle": {
+ "mode": "off"
+ }
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ }
+ ]
+ },
+ "unit": "short"
+ },
+ "overrides": [
+ {
+ "matcher": {
+ "id": "byName",
+ "options": "max blocked_ms"
+ },
+ "properties": [
+ {
+ "id": "custom.axisPlacement",
+ "value": "right"
+ },
+ {
+ "id": "unit",
+ "value": "ms"
+ }
+ ]
+ }
+ ]
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 24,
+ "x": 0,
+ "y": 45
+ },
+ "id": 23,
+ "options": {
+ "legend": {
+ "calcs": [
+ "mean",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "sum(count_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"wedge\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" [$__interval]))",
+ "legendFormat": "wedges",
+ "queryType": "range",
+ "refId": "A"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "max_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"wedge\" | matrix_user_id=~\"$mat |
Contributor
Preview deploymentsHost Test Results 1 files ±0 1 suites ±0 3h 3m 0s ⏱️ - 5m 46s Results for commit fb1e7f8. ± Comparison against earlier commit 23d02be. Realm Server Test Results 1 files ±0 1 suites ±0 9m 59s ⏱️ -45s Results for commit fb1e7f8. ± Comparison against earlier commit 23d02be. |
Grafana names an instant metric query's numeric column Value #<refId>; an organize transformation renames it to what it measures (wedges, events, max settle_ms, ...) and hides the redundant per-row timestamp. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 754b7a17a6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The images are referenced from the PR description via commit-pinned raw URLs; the working tree no longer needs to carry them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an always-on client performance telemetry pipeline end-to-end: a host-side instrument that batches events and POSTs them to realm-server, a realm-server ingest route that re-emits events as structured log lines for Loki, and a Grafana dashboard to explore the resulting signal.
Changes:
- Add
/_client-telemetryrealm-server route + handler that authenticates via JWT and logs one JSON line per event to theboxel:client-perfchannel. - Introduce a host
client-telemetryservice (eagerly started) plus hooks in network/store/loader paths to emit server-request, deserialize, rebuild, realm-event, card-load, wedge, and keepalive events. - Add a Loki-backed Grafana dashboard for client performance drilldowns, plus enable
Document-Policy: js-profilingin realm-server HTML responses and host dev servers.
Reviewed changes
Copilot reviewed 10 out of 11 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/realm-server/routes.ts | Wires a new authenticated POST /_client-telemetry route. |
| packages/realm-server/handlers/serve-index.ts | Adds Document-Policy: js-profiling on index HTML responses. |
| packages/realm-server/handlers/handle-client-telemetry.ts | Implements ingest: validates envelope and logs each event as a JSON line. |
| packages/observability/grafanactl/resources/dashboards/boxel-status/client-performance.json | Adds a new “Client Performance” Grafana dashboard using Loki queries over structured logs. |
| packages/host/vite.config.mjs | Enables Document-Policy: js-profiling in Vite dev/preview headers. |
| packages/host/tests/integration/client-telemetry-test.ts | Adds integration tests for the telemetry service’s gating and event shapes. |
| packages/host/app/services/store.ts | Emits deserialize + realm-event/rebuild telemetry from store lifecycle and invalidation handling. |
| packages/host/app/services/network.ts | Adds server-request timing middleware into authedFetch. |
| packages/host/app/services/client-telemetry.ts | New telemetry service: buffering, flush loop, wedge detection, optional JS self-profiler sampling, and request timing middleware. |
| packages/host/app/lib/gc-card-store.ts | Extends recent card doc load diagnostics with outcome for telemetry attribution. |
| packages/host/app/instance-initializers/register-client-telemetry.ts | Eagerly boots and tears down telemetry at app-instance lifecycle boundaries. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Cap each flush under the ~64KB fetch(keepalive) body budget, measured in UTF-8
bytes (a chunk between 64KB and the old 200KB char cap was silently rejected
by the browser and its events dropped).
- Reject oversized telemetry on the declared Content-Length before buffering the
whole request body; keep the post-read byte check as a fallback.
- Dashboard: an empty session textbox now matches all sessions
(`session_id=~".*${session_id}.*"`) instead of only an empty id.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The deserialize event's doc_bytes was JSON.stringify(doc).length — an O(size) re-serialization of the whole included graph on the hot deserialize path, purely to size it. fetchJSON now stamps the response Content-Length on the doc (O(1), non-enumerable), and the telemetry hook reads that instead. duration_ms (the actual deserialize cost) is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Referenced from the PR description via a commit-pinned raw URL. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ender - Scope card-load `loaded_ids`/`slowest_loads` to the load window by stamping each recent doc-load with the load generation and filtering to generations past the window start, so a slow background reload from an earlier window is not misattributed to the card being opened. - Stamp a logging correlation id on the outgoing request in the server-request timing middleware when one is absent, so real-user beacons carry a correlation id that joins to server-side `realm:requests` logs (previously only minted for prerender traffic). - Defer the wedge emit to a macrotask and only emit when the heartbeat gap is corroborated by a LoAF/longtask observation in the window, suppressing spurious wedges from timer coalescing after the tab was backgrounded. - Race the card-load window against a watchdog so a stuck load cannot hold the window open for the tab's lifetime and suppress every later card-load. - Drop the prior 401 attempt from the buffer when a retried request succeeds, so a transient auth refresh does not read as a server error. - Skip the deserialize/rebuild/realm-event hooks inside a prerender tab. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Give each table panel's instant-query value column a descriptive name via an organize transform, so panels no longer render an opaque "Value #A" header. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Card-load windows aren't measured while the tab is hidden (the heartbeat returns early), so `#lastLoadGeneration` freezes there while background reloads advance `store.loadGeneration`. On resume the visibility handler reset the heartbeat clock but not the generation cursor, so the first visible heartbeat saw a large generation jump and opened a card-load window spanning the whole hidden period — attributing background reloads to the now-focused card with a near-zero settle time. Fast-forward the generation cursor on resume, mirroring the heartbeat-clock reset, so a hidden-period jump never opens a window. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The early size check rejected only an oversized *declared* length; a request
with no Content-Length yielded `Number('')` → 0, which passed the check, and
the body was then read fully into memory before the post-read byte check ran —
so a chunked or unbounded body bypassed the cap. A telemetry beacon is always a
materialized JSON string whose client declares a length, so require a finite
positive Content-Length up front (411 otherwise) and reject an oversized
declaration before reading. The post-read byte check stays as defense in depth.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A genuine main-thread freeze always leaves a LoAF/longtask entry, but that observer callback can be delivered a little after the heartbeat that detected the gap. Dropping an uncorroborated gap immediately could therefore discard a real freeze whose entry simply hadn't arrived yet. Re-check a couple of times across a short grace window before concluding the gap was an OS sleep/suspend wake — a real sleep runs no task and never corroborates, so it is still dropped, while a lagging observer entry now has time to land. Guard the deferred emit on the started flag so a pending re-check after teardown is a no-op. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Covers the security-critical surface: authentication (401 without a token), the one-JSON-line-per-event emission and 204 response, and the trusted-envelope ordering — the JWT-authenticated user and the fixed channel win over client-supplied per-event fields, so a beacon cannot spoof either. Also asserts the size caps (411 with no declared length, 413 over the byte cap) and the input-validation rejections (malformed JSON, missing events array, over the event-count cap). The emitted lines are captured by tapping the shared boxel:client-perf logger's methodFactory. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…meout' into cs-12288-client-perf-telemetry
habdelra
changed the base branch from
main
to
flaky-base-realm-fetch-header-timeout
July 24, 2026 15:02
burieberry
reviewed
Jul 24, 2026
burieberry
left a comment
burieberry
left a comment
Contributor
There was a problem hiding this comment.
Code review
Two findings after verifying each against the diff. (Four other candidates were investigated and dropped — a search-entries race, an index-writer "third full-table scan", an oversized-telemetry-event, and the full-reindex own_write label — none survived verification, either factually wrong or with severity overstated.)
🐛 persist-scroll-position.ts — momentum/inertial scroll offsets aren't persisted on touch devices
The restore-suppression gate swallows inertial scroll events, so on touch devices the persisted offset is the position a frame or two after finger-lift, not the final resting position. Reopening the sheet then restores to the wrong place after any flick-scroll.
The chain:
restoringis a plain local reset totrueon every modifier run (let restoring = true), andrecordis gated on!restoring.restoringis cleared tofalseonly bystopRestoring, which is attached exclusively to gesture events:['wheel', 'touchmove', 'pointerdown', 'keydown']. Thescrolllistener runsrecordand never clears it.record→onChange(scrollTop)writes back the trackedresultsScrollTopthat thescrollToparg reads, which re-runs the modifier and re-armsrestoring = true(as the header comment notes).
So after a flick: the last touchmove clears restoring, the first momentum scroll events record and trigger a write-back, the modifier re-installs with restoring = true, and every remaining momentum scroll (iOS inertia can run ~1s+) is dropped because momentum emits no further gesture event to clear the gate.
Keeping restoring = true on re-install is clearly deliberate — it suppresses a layout-induced scroll reset right after re-install. But it also discards the real momentum settle. Options:
- Only re-arm
restoringon a genuine re-install (first install / an actual restore-target change), not on a self-triggered write-back re-run. - Or clear
restoringon the firstscrollafter a re-install once the offset matches the restored target, so subsequent momentum scrolls record.
Either way, worth a test asserting a gesture-less scroll following a write-back re-install is recorded — the current suite only asserts the opposite (gesture-less scrolls ignored), which is why this slipped through.
ℹ️ Non-blocking — realm.ts file-meta serve now does an unconditional DB lookup
The (searchHash === undefined || searchSize === undefined) guard was dropped, so getContentMeta() (an uncached SELECT on realm_file_meta) now runs on every file-meta serve even when the indexed searchDoc already carries both values. It's a single indexed-PK lookup so the cost is small, and the staleness rationale is well-commented — just flagging that the added per-request query isn't mentioned in the comments. Fine to leave as-is if the round-trip is acceptable on that path.
🤖 Review generated with Claude Code
Contributor
Author
|
[Claude Code 🤖] Thanks for the careful pass. Both flagged files fall outside this change set — this branch modifies neither |
habdelra
changed the base branch from
flaky-base-realm-fetch-header-timeout
to
main
July 27, 2026 13:28
Resolve the factory-skill-loader pending-references test conflict in favor of main's single-deepEqual assertion form. Both branches independently restructured the same assertion; main's is the integration target and already carries an assertion in the empty-pending case, so its form wins. The base-realm fetch header-timeout fix (virtual-network.ts) and the telemetry work are unaffected.
lukemelia
approved these changes
Jul 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
The client had no performance signal in the field. When a user's tab froze on "Loading card…" or went unresponsive for minutes, there was nothing to diagnose it after the fact, and everyday client cost — server round-trips, response deserialization, card-to-interactive latency, loader rebuilds — was invisible. This adds an always-on, passive client performance instrument, an ingest endpoint, and a Grafana dashboard, so real client behavior is measurable and attributable to a specific user or session.
Pipeline: host instrument →
fetch(keepalive)POST/_client-telemetry→ structuredboxel:client-perflog line → alloy → Loki → Grafana (the same log pipeline every server log already uses — no new infrastructure).What it measures
Six event types:
realm:requests/realm:search-timinglogs via the logging correlation id.function @ source-url:char-offset(source-map-resolvable, so it's readable for minified builds) plus the worst few blocking scripts, and — on profiler-sampled sessions — real JS self-profiler call stacks.foo.gtschanged").Extending it — new telemetry is welcome
The six event types are a starting set, not a closed schema. The instrument, the ingest route, and the dashboard are all event-type-agnostic: the route re-emits whatever event objects it's handed as
boxel:client-perfJSON lines, and the dashboard reads fields at query time with| json. Adding a new kind of client signal is a small, local change — define the event shape, record it from the relevant hook, and add a panel that reads its fields. Devs should feel welcome to instrument other parts of the client this way: Matrix sync latency and reconnect churn, AI assistant round-trips and streaming responsiveness, command execution timings — anything where the client's real behavior is otherwise invisible in the field.Sampling cadence vs. network cadence (decoupled)
Stats are gathered at their natural cadence, independent of when the client talks to the server:
pagehide, wherekeepalivelets the last POST complete, capped under the browser's ~64KB keepalive body budget).Design (live invariants)
enableForTest) and inside a prerender tab; every web API is feature-detected; every timer / observer / listener is torn down.env,service) are Loki stream labels;session_id/matrix_user_id/card_id/urlride inside the JSON line and are extracted at query time with| json, so per-user/per-session drill-down stays possible without exploding Loki's index.Dashboard
An overview row of headline health, a collapsible row per event type, a by-user row that surfaces which accounts are hitting issues (wedges, worst card-to-interactive, and signal volume per user), and a wedge breakdown that names the blocking frame and lists the worst blocking scripts per wedge. Shared
env/matrix_user_id/session_idtemplate variables drill the whole board into a single client. Works in local dev via the observabilitydocker-composestack +./scripts/apply.sh --env local.Screenshots below are the local dashboard driven by synthetic telemetry.
Overview + per-event-type rows
By user — who is hitting issues
Recent wedges with their blocking scripts (
fn @ url:char, source-map-resolvable)Testing
/_client-telemetryroute caps body size (early on Content-Length, then a byte fallback) and event count, authenticates viajwtMiddleware, and emits one JSON line per event. Type + lint clean.| json/unwrap/quantile_over_timequeries, per-user drill-down, and wedge-frame panel all render.Follow-ups to consider
Not committed to in this PR — options worth weighing later:
server-requesttelemetry volume if the Loki ingestion cost turns out to matter at scale.🤖 Generated with Claude Code